{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9080b39e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import sys\n",
    "\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
    "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8d0b2364-4806-4656-81e7-3f6e4b910b5b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jerryliu/Programming/gpt_index_forks/gpt_index_ravi/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from gpt_index import (\n",
    "    GPTTreeIndex, \n",
    "    GPTSimpleVectorIndex, \n",
    "    SimpleDirectoryReader, \n",
    "    LLMPredictor, \n",
    "    ServiceContext\n",
    ")\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.llms import OpenAI\n",
    "from gpt_index.evaluation import ResponseEvaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b9b98f89-d5b8-4d29-92f6-ad76d5060e9f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# gpt-3 (davinci)\n",
    "llm_predictor_gpt3 = LLMPredictor(llm=OpenAI(temperature=0, model_name=\"text-davinci-003\"))\n",
    "service_context_gpt3 = ServiceContext.from_defaults(llm_predictor=llm_predictor_gpt3)\n",
    "\n",
    "# gpt-4\n",
    "llm_predictor_gpt4 = LLMPredictor(llm=ChatOpenAI(temperature=0, model_name=\"gpt-4\"))\n",
    "service_context_gpt4 = ServiceContext.from_defaults(llm_predictor=llm_predictor_gpt4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8eb3e616-64e5-4bf4-a67b-661e9b3657e7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "evaluator = ResponseEvaluator(service_context=service_context_gpt3)\n",
    "evaluator_gpt4 = ResponseEvaluator(service_context=service_context_gpt4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1298bbb4-c99e-431e-93ef-eb32c0a2fc2a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "documents = SimpleDirectoryReader('../test_wiki/data').load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dca06a5b-8a15-40b4-8c7f-dae5407c674f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load tree index from stored index\n",
    "tree_index = GPTTreeIndex.load_from_disk('../test_wiki/index.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41f0e53f-77a6-40d5-94ae-3f81b01af75c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create vector index\n",
    "vector_index = GPTSimpleVectorIndex.from_documents(\n",
    "    documents, \n",
    "    service_context=ServiceContext.from_defaults(chunk_size_limit=512)\n",
    ")\n",
    "vector_index.save_to_disk('../test_wiki/simple_vector_index.json')\n",
    "\n",
    "vector_index = GPTSimpleVectorIndex.load_from_disk('../test_wiki/simple_vector_index.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68c9ebfe-b1b6-4f4e-9278-174346de8c90",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response_tree = tree_index.query(\"What battles took place in New York City in the American Revolution?\")\n",
    "eval_result = evaluator_gpt4.evaluate(response_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "44311717-3077-4acb-9ece-2a14ba0125a3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Response: The Battle of Long Island, the Battle of White Plains, the Battle of Harlem Heights, the Battle of Fort Washington, the Battle of Fort Lee, and the Battle of Yorktown all took place in New York City during the American Revolution. These battles took place in various locations throughout the city, including Battery Weed and Fort Tompkins, Great Kills Park, and Central Park.\n",
      "\n",
      "> Source: in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the borough of Manhattan, including Randalls and Wards Islands, and Roosevelt Island in the East River, and Governors Island and Liberty Island to the south in New York Harbor.\n",
      "Manhattan Island is loosely divided into the Lower, Midtown, and Uptown regions. Uptown Manhattan is divided by Central Park into the Upper East Side and the Upper West Side, and above the park is Harlem, bordering the Bronx (Bronx County).\n",
      "Harlem was predominantly occupied by Jewish and Italian Americans in the 19th century until the Great Migration. It was the center of the Harlem Renaissance.\n",
      "\n",
      "The borough of Manhattan also includes a small neighborhood on the mainland, called Marble Hill, which is contiguous with the Bronx. New York City's remaining four boroughs are collectively referred to as the Outer Boroughs.\n",
      "\n",
      "\n",
      "==== Brooklyn ====\n",
      "Brooklyn (Kings County), on the western tip of Long Island, is the city's most populous borough. Brooklyn is known for its cultural, social, and ethnic diversity, an independent art scene, distinct neighborhoods, and a distinctive architectural heritage. Downtown Brooklyn is the largest central core neighborhood in the Outer Boroughs. The borough has a long beachfront shoreline including Coney Island, established in the 1870s as one of the earliest amusement grounds in the U.S. Marine Park and Prospect Park are the two largest parks in Brooklyn. Since 2010, Brooklyn has evolved into a thriving hub of entrepreneurship and high technology startup firms, and of postmodern art and design.\n",
      "\n",
      "\n",
      "==== Queens ====\n",
      "Queens (Queens County), on Long Island north and east of Brooklyn, is geographically the largest borough, the most ethnically diverse county in the United States, and the most ethnically diverse urban area in the world. Historically a collection of small towns and villages founded by the Dutch, the borough has since developed both commercial and residential prominence. Downtown Flushing has become one of the busiest central core neighborhoods in the outer boroughs. Queens is the site of the Citi Field baseball stadium, home of the New York Mets, and hosts the annual U.S. Open tennis tournament at Flushing Meadows–Corona Park. Additionally, two of the three busiest airports serving the New York metropolitan area, John F. Kennedy International Airport and LaGuardia Airport, are in Queens. The third is Newark Liberty International Airport in Newark, New Jersey.\n",
      "\n",
      "\n",
      "==== The Bronx ====\n",
      "The Bronx (Bronx County) is both New York City's northernmost borough, and the only one that is mostly on the mainland. It is the location of Yankee Stadium, the baseball park of the New York Yankees, and home to the largest cooperatively-owned housing complex in the United States, Co-op City. It is also home to the Bronx Zoo, the world's largest metropolitan zoo, which spans 265 acres (1.07 km2) and houses more than 6,000 animals. The Bronx is also the birthplace of hip hop music and its associated culture. Pelham Bay Park is the largest park in New York City, at 2,772 acres (1,122 ha).\n",
      "\n",
      "\n",
      "==== Staten Island ====\n",
      "Staten Island (Richmond County) is the most suburban in character of the five boroughs. Staten Island is connected to Brooklyn by the Verrazzano-Narrows Bridge, and to Manhattan by way of the free Staten Island Ferry, a daily commuter ferry that provides unobstructed views of the Statue of Liberty, Ellis Island, and Lower Manhattan. In central Staten Island, the Staten Island Greenbelt spans approximately 2,500 acres (10 km2), including 28 miles (45 km) of walking trails and one of the last undisturbed forests in the city. Designated in 1984 to protect the island's natural lands, the Greenbelt comprises seven city parks.\n",
      "\n",
      "\t\t\n",
      "\t\t\n",
      "\n",
      "\n",
      "=== Architecture ===\n",
      "\n",
      "New York has architecturally noteworthy buildings in a wide range of styles and from distinct time periods, from the Dutch Colonial Pieter Claesen Wyckoff House in Brooklyn, the oldest section of which dates to 1656, to the modern One World Trade Center, the skyscraper at Ground Zero in Lower Manhattan and the most expensive office tower in the world by construction cost.Manhattan's skyline, with its many skyscrapers, is universally recognized, and the city has been home to several of the tallest buildings in the world. As of 2019, New York City had 6,455 high-rise buildings, the third most in the world after Hong Kong and Seoul. Of these, as of 2011, 550 completed structures were at least 330 feet (100 m) high, with more than fifty completed skyscrapers taller than 656 feet (200 m). These include the Woolworth Building, an early example of Gothic Revival architecture in skyscraper design, built with massively scaled Gothic detailing; completed in 1913, for 17 years it was the world's tallest building.The 1916 Zoning Resolution required setbacks in new buildings and restricted towers to a percentage of the lot size, to allow sunlight to reach the streets below. The Art Deco style of the Chrysler Building (1930) and Empire State Building (1931), with their tapered tops and steel spires, reflected the zoning requirements. The buildings have distinctive ornamentation, such as the eagles at the corners of the 61st floor on the Chrysler Building, and are considered some of the finest examples of the Art Deco style. A highly influential example of the International Style in the United States is the Seagram Building (1957), distinctive for its façade using visible bronze-toned I-beams to evoke the building's structure. The Condé Nast Building (2000) is a prominent example of green design in American skyscrapers and has received an award from the American Institute of Architects and AIA New York State for its design.\n",
      "The character of New York's large residential districts is often defined by the elegant brownstone rowhouses and townhouses and shabby tenements that were built during a period of rapid expansion from 1870 to 1930. In contrast, New York City also has neighborhoods that are less densely populated and feature free-standing dwellings. In neighborhoods such as Riverdale (in the Bronx), Ditmas Park (in Brooklyn), and Douglaston (in Queens), large single-family homes are common in various architectural styles such as Tudor Revival and Victorian.Stone and brick became the city's building materials of choice after the construction of wood-frame houses was limited in the aftermath of the Great Fire of 1835. A distinctive feature of many of the city's buildings is the roof-mounted wooden water tower. In the 1800s, the city required their installation on buildings higher than six stories to prevent the need for excessively high water pressures at lower elevations, which could break municipal water pipes. Garden apartments became popular during the 1920s in outlying areas, such as Jackson Heights.According to the United States Geological Survey, an updated analysis of seismic hazard in July 2014 revealed a \"slightly lower hazard for tall buildings\" in New York City than previously assessed. Scientists estimated this lessened risk based upon a lower likelihood than previously thought of slow shaking near the city, which would be more likely to cause damage to taller structures from an earthquake in the vicinity of the city. Manhattan contained over 500 million square feet of office space as of 2022; the COVID-19 pandemic and hybrid work model have prompted consideration of commercial-to-residential conversion within Midtown Manhattan.\n",
      "\n",
      "\n",
      "=== Climate ===\n",
      "\n",
      "Under the Köppen climate classification, using the 0 °C (32 °F) isotherm, New York City features a humid subtropical climate (Cfa), and is thus the northernmost major city on the North American continent with this categorization. The suburbs to the immediate north and west lie in the transitional zone between humid subtropical and humid continental climates (Dfa). By the Trewartha classification, the city is defined as having an oceanic climate (Do). Annually, the city averages 234 days with at least some sunshine. The city lies in the USDA 7b plant hardiness zone.Winters are chilly and damp, and prevailing wind patterns that blow sea breezes offshore temper the moderating effects of the Atlantic Ocean; yet the Atlantic and the partial shielding from colder air by the Appalachian Mountains keep the city warmer in the winter than inland North American cities at similar or lesser latitudes such as Pittsburgh, Cincinnati, and Indianapolis. The daily mean temperature in January, the area's coldest month, is 33.3 °F (0.7 °C). Temperatures usually drop to 10 °F (−12 °C) several times per winter, yet can also reach 60 °F (16 °C) for several days even in the coldest winter month. Spring and autumn are unpredictable and can range from cool to warm, although they are usually mild with low humidity. Summers are typically hot and humid, with a daily mean temperature of 77.5 °F (25.3 °C) in July.Nighttime temperatures are often enhanced due to the urban heat island effect. Daytime temperatures exceed 90 °F (32 °C) on average of 17 days each summer and in some years exceed 100 °F (38 °C), although this is a rare achievement, last occurring on July 18, 2012. Similarly, readings of 0 °F (−18 °C) are also extremely rare, last occurring on February 14, 2016. Extreme temperatures have ranged from −15 °F (−26 °C), recorded on February 9, 1934, up to 106 °F (41 °C) on July 9, 1936; the coldest recorded wind chill was −37 °F (−38 °C) on the same day as the all-time record low. The record cold daily maximum was 2 °F (−17 °C) on December 30, 1917, while, conversely, the record warm daily minimum was 87 °F (31 °C), on July 2, 1903. The average water temperature of the nearby Atlantic Ocean ranges from 39.7 °F (4.3 °C) in February to 74.1 °F (23.4 °C) in August.The city receives 49.5 inches (1,260 mm) of precipitation annually, which is relatively evenly spread throughout the year. Average winter snowfall between 1991 and 2020 has been 29.8 inches (76 cm); this varies considerably between years. Hurricanes and tropical storms are rare in the New York area. Hurricane Sandy brought a destructive storm surge to New York City on the evening of October 29, 2012, flooding numerous streets, tunnels, and subway lines in Lower Manhattan and other areas of the city and cutting off electricity in many parts of the city and its suburbs. The storm and its profound impacts have prompted the discussion of constructing seawalls and other coastal barriers around the shorelines of the city and the metropolitan area to minimize the risk of destructive consequences from another such event in the future.The coldest month on record is January 1857, with a mean temperature of 19.6 °F (−6.9 °C) whereas the warmest months on record are July 1825 and July 1999, both with a mean temperature of 81.4 °F (27.4 °C). The warmest years on record are 2012 and 2020, both with mean temperatures of 57.1 °F (13.9 °C). The coldest year is 1836, with a mean temperature of 47.3 °F (8.5 °C). The driest month on record is June 1949, with 0.02 inches (0.51 mm) of rainfall. The wettest month was August 2011, with 18.95 inches (481 mm) of rainfall. The driest year on record is 1965, with 26.09 inches (663 mm) of rainfall. The wettest year was 1983, with 80.56 inches (2,046 mm) of rainfall. The snowiest month on record is February 2010, with 36.9 inches (94 cm) of snowfall. The snowiest season (Jul–Jun) on record is 1995–1996, with 75.6 inches (192 cm) of snowfall. The least snowy season was 1972–1973, with 2.3 inches (5.8 cm) of snowfall. The earliest seasonal trace of snowfall occurred on October 10, in both 1979 and 1925. The latest seasonal trace of snowfall occurred on May 9, in both 2020 and 1977.\n",
      "\n",
      "See or edit raw graph data.\n",
      "\n",
      "\n",
      "=== Parks ===\n",
      "\n",
      "The city of New York has a complex park system, with various lands operated by the National Park Service, the New York State Office of Parks, Recreation and Historic Preservation, and the New York City Department of Parks and Recreation. In its 2018 ParkScore ranking, the Trust for Public Land reported that the park system in New York City was the ninth-best park system among the fifty most populous U.S. cities. ParkScore ranks urban park systems by a formula that analyzes median park size, park acres as percent of city area, the percent of city residents within a half-mile of a park, spending of park services per resident, and the number of playgrounds per 10,000 residents. In 2021, the New York City Council banned the use of synthetic pesticides by city agencies and instead required organic lawn management. The effort was started by teacher Paula Rogovin's kindergarten class at P.S. 290.\n",
      "\n",
      "\n",
      "==== National parks ====\n",
      "\n",
      "Gateway National Recreation Area contains over 26,000 acres (110 km2), most of it in New York City. In Brooklyn and Queens, the park contains over 9,000 acres (36 km2) of salt marsh, wetlands, islands, and water, including most of Jamaica Bay and the Jamaica Bay Wildlife Refuge. Also in Queens, the park includes a significant portion of the western Rockaway Peninsula, most notably Jacob Riis Park and Fort Tilden. In Staten Island, it includes Fort Wadsworth, with historic pre-Civil War era Battery Weed and Fort Tompkins, and Great Kills Park, with beaches, trails, and a marina.\n",
      "The Statue of Liberty National Monument and Ellis Island Immigration Museum are managed by the National Park Service and are in both New York and New Jersey. They are joined in the harbor by Governors Island National Monument. Historic sites under federal management on Manhattan Island include Stonewall National Monument; Castle Clinton National Monument; Federal Hall National Memorial; Theodore Roosevelt Birthplace National Historic Site; General Grant National Memorial (Grant's Tomb); African Burial Ground National Monument; and Hamilton Grange National Memorial. Hundreds of properties are listed on the National Register of Historic Places or as a National Historic Landmark.\n",
      "\n",
      "\n",
      "==== State parks ====\n",
      "\n",
      "There are seven state parks within the confines of New York City. Some of them include:\n",
      "\n",
      "The Clay Pit Ponds State Park Preserve is a natural area that includes extensive riding trails.\n",
      "Riverbank State Park is a 28-acre (11 ha) facility that rises 69 feet (21 m) over the Hudson River.\n",
      "Marsha P. Johnson State Park is a state park in Brooklyn and Manhattan that borders the East River that was renamed in honor of Marsha P. Johnson.\n",
      "\n",
      "\n",
      "==== City parks ====\n",
      "\n",
      "New York City has over 28,000 acres (110 km2) of municipal parkland and 14 miles (23 km) of public beaches. The largest municipal park in the city is Pelham Bay Park in the Bronx, with 2,772 acres (1,122 ha).\n",
      "Central Park, an 843-acre (3.41 km2) park in middle-upper\n",
      "\n",
      "> Evaluation Result: NO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    f\"> Response: {response_tree}\\n\\n\"\n",
    "    f\"> Source: {response_tree.source_nodes[0].source_text}\\n\\n\"\n",
    "    f\"> Evaluation Result: {eval_result}\\n\\n\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "180a5d2e-9286-477b-9cd0-a5976d18d845",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:gpt_index.token_counter.token_counter:> [query] Total LLM token usage: 476 tokens\n",
      "> [query] Total LLM token usage: 476 tokens\n",
      "INFO:gpt_index.token_counter.token_counter:> [query] Total embedding token usage: 13 tokens\n",
      "> [query] Total embedding token usage: 13 tokens\n",
      "INFO:gpt_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens\n",
      "> [build_index_from_nodes] Total LLM token usage: 0 tokens\n",
      "INFO:gpt_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens\n",
      "> [build_index_from_nodes] Total embedding token usage: 0 tokens\n",
      "INFO:gpt_index.token_counter.token_counter:> [query] Total LLM token usage: 691 tokens\n",
      "> [query] Total LLM token usage: 691 tokens\n",
      "INFO:gpt_index.token_counter.token_counter:> [query] Total embedding token usage: 0 tokens\n",
      "> [query] Total embedding token usage: 0 tokens\n"
     ]
    }
   ],
   "source": [
    "response_vector = vector_index.query(\"What battles took place in New York City in the American Revolution?\")\n",
    "eval_result = evaluator.evaluate(response_vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c764b8b3-69b1-4ac8-b88b-3f9e204b8bfb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Response: \n",
      "The Battle of Long Island and the Great Fire of New York.\n",
      "\n",
      "> Source: at labor. Slavery became integrally tied to New York's economy through the labor of slaves throughout the port, and the banking and shipping industries trading with the American South. During construction in Foley Square in the 1990s, the African Burying Ground was discovered; the cemetery included 10,000 to 20,000 of graves of colonial-era Africans, some enslaved and some free.The 1735 trial and acquittal in Manhattan of John Peter Zenger, who had been accused of seditious libel after criticizing colonial governor William Cosby, helped to establish the freedom of the press in North America. In 1754, Columbia University was founded under charter by King George II as King's College in Lower Manhattan.\n",
      "\n",
      "\n",
      "=== American Revolution ===\n",
      "\n",
      "The Stamp Act Congress met in New York in October 1765, as the Sons of Liberty organization emerged in the city and skirmished over the next ten years with British troops stationed there. The Battle of Long Island, the largest battle of the American Revolutionary War, was fought in August 1776 within the modern-day borough of Brooklyn. After the battle, in which the Americans were defeated, the British made the city their military and political base of operations in North America. The city was a haven for Loyalist refugees and escaped slaves who joined the British lines for freedom newly promised by the Crown for all fighters. As many as 10,000 escaped slaves crowded into the city during the British occupation. When the British forces evacuated at the close of the war in 1783, they transported 3,000 freedmen for resettlement in Nova Scotia. They resettled other freedmen in England and the Caribbean.\n",
      "The only attempt at a peaceful solution to the war took place at the Conference House on Staten Island between American delegates, including Benjamin Franklin, and British general Lord Howe on September 11, 1776. Shortly after the British occupation began, the Great Fire of New York occurred, a large conflagration on the West Side of Lower Manhattan, which destroyed about a quarter of the buildings in the city, including Trinity Church.In 1785, the assembly of\n",
      "\n",
      "> Evaluation Result:  NO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    f\"> Response: {response_vector}\\n\\n\"\n",
    "    f\"> Source: {response_vector.source_nodes[0].source_text}\\n\\n\"\n",
    "    f\"> Evaluation Result: {eval_result}\\n\\n\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4fc3f18a-0ef9-453c-acf8-7aedd784cdcf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:gpt_index.indices.query.tree.leaf_query:> Starting query: What are the airports in New York City?\n",
      "> Starting query: What are the airports in New York City?\n",
      "INFO:gpt_index.indices.query.tree.leaf_query:>[Level 1] Selected node: [9]/[9]\n",
      ">[Level 1] Selected node: [9]/[9]\n",
      "INFO:gpt_index.token_counter.token_counter:> [query] Total LLM token usage: 4846 tokens\n",
      "> [query] Total LLM token usage: 4846 tokens\n",
      "INFO:gpt_index.token_counter.token_counter:> [query] Total embedding token usage: 0 tokens\n",
      "> [query] Total embedding token usage: 0 tokens\n",
      "INFO:gpt_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens\n",
      "> [build_index_from_nodes] Total LLM token usage: 0 tokens\n",
      "INFO:gpt_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens\n",
      "> [build_index_from_nodes] Total embedding token usage: 0 tokens\n",
      "INFO:gpt_index.token_counter.token_counter:> [query] Total LLM token usage: 1788 tokens\n",
      "> [query] Total LLM token usage: 1788 tokens\n",
      "INFO:gpt_index.token_counter.token_counter:> [query] Total embedding token usage: 0 tokens\n",
      "> [query] Total embedding token usage: 0 tokens\n"
     ]
    }
   ],
   "source": [
    "response_tree = tree_index.query(\"What are the airports in New York City?\")\n",
    "eval_result = evaluator.evaluate(response_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a34490f6-7242-4c31-b49e-b2a65d9923ab",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Response: The airports in New York City are John F. Kennedy International Airport (JFK), LaGuardia Airport (LGA), and Newark Liberty International Airport (EWR).\n",
      "\n",
      "> Source: Avenue are also used as metonyms for national industries there: the theater, finance, advertising, and fashion organizations, respectively.\n",
      "New York City also has an extensive web of freeways and parkways, which link the city's boroughs to each other and to North Jersey, Westchester County, Long Island, and southwestern Connecticut through various bridges and tunnels. Because these highways serve millions of outer borough and suburban residents who commute into Manhattan, it is quite common for motorists to be stranded for hours in traffic congestion that are a daily occurrence, particularly during rush hour. Congestion pricing in New York City will go into effect in 2022 at the earliest.New York City is also known for its rules regarding turning at red lights. Unlike the rest of the United States, New York State prohibits right or left turns on red in cities with a population greater than one million, to reduce traffic collisions and increase pedestrian safety. In New York City, therefore, all turns at red lights are illegal unless a sign permitting such maneuvers is present.\n",
      "\n",
      "\n",
      "==== River crossings ====\n",
      "\n",
      "New York City is located on one of the world's largest natural harbors, and the boroughs of Manhattan and Staten Island are primarily coterminous with islands of the same names, while Queens and Brooklyn are at the west end of the larger Long Island, and the Bronx is on New York State's mainland. This situation of boroughs separated by water led to the development of an extensive infrastructure of bridges and tunnels.\n",
      "The George Washington Bridge is the world's busiest motor vehicle bridge, connecting Manhattan to Bergen County, New Jersey. The Verrazzano-Narrows Bridge is the longest suspension bridge in the Americas and one of the world's longest. The Brooklyn Bridge is an icon of the city itself. The towers of the Brooklyn Bridge are built of limestone, granite, and Rosendale cement, and their architectural style is neo-Gothic, with characteristic pointed arches above the passageways through the stone towers. This bridge was also the longest suspension bridge in the world from its opening until 1903, and is the first steel-wire suspension bridge. The Queensboro Bridge is an important piece of cantilever architecture. The Manhattan Bridge, opened in 1909, is considered to be the forerunner of modern suspension bridges, and its design served as the model for many of the long-span suspension bridges around the world; the Manhattan Bridge, Throgs Neck Bridge, Triborough Bridge, and Verrazano-Narrows Bridge are all examples of structural expressionism.Manhattan Island is linked to New York City's outer boroughs and to New Jersey. The Lincoln Tunnel, which carries 120,000 vehicles a day under the Hudson River between New Jersey and Midtown Manhattan, is the busiest vehicular tunnel in the world. The tunnel was built instead of a bridge to allow unfettered passage of large passenger and cargo ships that sailed through New York Harbor and up the Hudson River to Manhattan's piers. The Holland Tunnel, connecting Lower Manhattan to Jersey City, New Jersey, was the world's first mechanically ventilated vehicular tunnel when it opened in 1927. The Queens–Midtown Tunnel, built to relieve congestion on the bridges connecting Manhattan with Queens and Brooklyn, was the largest non-federal project in its time when it was completed in 1940. President Franklin D. Roosevelt was the first person to drive through it. The Brooklyn–Battery Tunnel (officially known as the Hugh L. Carey Tunnel) runs underneath Battery Park and connects the Financial District at the southern tip of Manhattan to Red Hook in Brooklyn.\n",
      "\n",
      "\n",
      "=== Cycling network ===\n",
      "\n",
      "Cycling in New York City is associated with mixed cycling conditions that include urban density, relatively flat terrain, congested roadways with stop-and-go traffic, and many pedestrians. The city's large cycling population includes utility cyclists, such as delivery and messenger services; cycling clubs for recreational cyclists; and an increasing number of commuters. Cycling is increasingly popular in New York City; in 2017 there were approximately 450,000 daily bike trips, compared with 170,000 daily bike trips in 2005. As of 2017, New York City had 1,333 miles (2,145 km) of bike lanes, compared to 513 miles (826 km) of bike lanes in 2006. As of 2019, there are 126 miles (203 km) of segregated or \"protected\" bike lanes citywide.\n",
      "\n",
      "\n",
      "== People ==\n",
      "\n",
      "\n",
      "== Global outreach ==\n",
      "\n",
      "In 2006, the Sister City Program of the City of New York, Inc. was restructured and renamed New York City Global Partners. Through this program, New York City has expanded its international outreach to a network of cities worldwide, promoting the exchange of ideas and innovation between their citizenry and policymakers. New York's historic sister cities are denoted below by the year they joined New York City's partnership network.\n",
      "\n",
      "\n",
      "== See also ==\n",
      "Outline of New York City\n",
      "\n",
      "\n",
      "== Notes ==\n",
      "\n",
      "\n",
      "== References ==\n",
      "\n",
      "\n",
      "== Further reading ==\n",
      "Belden, E. Porter (1849). New York, Past, Present, and Future: Comprising a History of the City of New York, a Description of its Present Condition, and an Estimate of its Future Increase. New York: G.P. Putnam. From Google Books.\n",
      "Burgess, Anthony (1976). New York. New York: Little, Brown & Co. ISBN 978-90-6182-266-0.\n",
      "Burrows, Edwin G. and Wallace, Mike (1999). Gotham: A History of New York City to 1898. New York: Oxford University Press. ISBN 0-195-11634-8.\n",
      "Federal Writers' Project (1939). The WPA Guide to New York City (1995 reissue ed.). New York: The New Press. ISBN 978-1-56584-321-9.\n",
      "Jackson, Kenneth T., ed. (1995). The Encyclopedia of New York City. New Haven: Yale University Press. ISBN 0300055366.\n",
      "Jackson, Kenneth T.; Dunbar, David S., eds. (2005). Empire City: New York Through the Centuries. Columbia University Press. ISBN 978-0-231-10909-3.\n",
      "Lankevich, George L. (1998). American Metropolis: A History of New York City. NYU Press. ISBN 978-0-8147-5186-2.\n",
      "White, E.B. (1949). Here is New York (2000 reissue ed.). Little Bookroom.\n",
      "White, Norval & Willensky, Elliot (2000). AIA Guide to New York City (4th ed.). New York: Three Rivers Press. ISBN 978-0-8129-3107-5.\n",
      "Whitehead, Colson (2003). The Colossus of New York: A City in 13 Parts. New York: Doubleday. ISBN 978-0-385-50794-3.\n",
      "\n",
      "\n",
      "== External links ==\n",
      "\n",
      "Official website \n",
      "NYC Go, official tourism website\n",
      "New York City at Curlie\n",
      " Geographic data related to New York City at OpenStreetMap\n",
      "Collections, 145,000 NYC photographs at the Museum of the City of New York\n",
      "\"The New New York Skyline (interactive)\". National Geographic. November 2015.\n",
      "\n",
      "> Evaluation Result:  NO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    f\"> Response: {response_tree}\\n\\n\"\n",
    "    f\"> Source: {response_tree.source_nodes[0].source_text}\\n\\n\"\n",
    "    f\"> Evaluation Result: {eval_result}\\n\\n\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97f3ddf1-8dc2-4fb8-831f-2c06649e0955",
   "metadata": {},
   "outputs": [],
   "source": [
    "response_vector = vector_index.query(\"What are the airports in New York City?\")\n",
    "eval_result = evaluator.evaluate(response_vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "01c53014-82b0-4865-b849-c0beb042143d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Response: \n",
      "The airports in New York City are Long Island MacArthur Airport, Trenton–Mercer Airport, Westchester County Airport, and Teterboro Airport.\n",
      "\n",
      "> Source: announced in July 2015 to entirely rebuild LaGuardia Airport in a multibillion-dollar project to replace its aging facilities. Other commercial airports in or serving the New York metropolitan area include Long Island MacArthur Airport, Trenton–Mercer Airport and Westchester County Airport. The primary general aviation airport serving the area is Teterboro Airport.\n",
      "\n",
      "\n",
      "=== Ferries ===\n",
      "\n",
      "The Staten Island Ferry is the world's busiest ferry route, carrying more than 23 million passengers from July 2015 through June 2016 on the 5.2-mile (8.4 km) route between Staten Island and Lower Manhattan and running 24 hours a day. Other ferry systems shuttle commuters between Manhattan and other locales within the city and the metropolitan area.\n",
      "NYC Ferry, a NYCEDC initiative with routes planned to travel to all five boroughs, was launched in 2017, with second graders choosing the names of the ferries. Meanwhile, Seastreak ferry announced construction of a 600-passenger high-speed luxury ferry in September 2016, to shuttle riders between the Jersey Shore and Manhattan, anticipated to start service in 2017; this would be the largest vessel in its class.\n",
      "\n",
      "\n",
      "=== Taxis, vehicles for hire, and trams ===\n",
      "\n",
      "Other features of the city's transportation infrastructure encompass 13,587 yellow taxicabs; other vehicle for hire companies; and the Roosevelt Island Tramway, an aerial tramway that transports commuters between Roosevelt Island and Manhattan Island.\n",
      "\n",
      "\n",
      "=== Streets and highways ===\n",
      "\n",
      "Despite New York's heavy reliance on its vast public transit system, streets are a defining feature of the city. The Commissioners' Plan of 1811 greatly influenced the city's physical development. Several of the city's streets and avenues, including Broadway, Wall Street, Madison Avenue, and Seventh Avenue are also used as metonyms for national industries there: the theater, finance, advertising, and fashion organizations, respectively.\n",
      "New York City also has an extensive web of freeways and parkways, which link the city's boroughs to each other and to North Jersey,\n",
      "\n",
      "> Evaluation Result:  NO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    f\"> Response: {response_vector}\\n\\n\"\n",
    "    f\"> Source: {response_vector.source_nodes[0].source_text}\\n\\n\"\n",
    "    f\"> Evaluation Result: {eval_result}\\n\\n\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b860691-9f6a-4b2f-bfef-5a5a56693a18",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response_vector = vector_index.query(\"Who is the mayor of New York City?\")\n",
    "eval_result = evaluator.evaluate(response_vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "91cce4e3-b4e5-4583-9543-6c3726c0e7d3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Response: \n",
      "The mayor of New York City is Eric Adams.\n",
      "\n",
      "> Source: by geographic population boundaries. Each term for the mayor and council members lasts four years and has a two consecutive-term limit, which is reset after a four-year break. The New York City Administrative Code, the New York City Rules, and the City Record are the code of local laws, compilation of regulations, and official journal, respectively.Each borough is coextensive with a judicial district of the state Unified Court System, of which the Criminal Court and the Civil Court are the local courts, while the New York Supreme Court conducts major trials and appeals. Manhattan hosts the First Department of the Supreme Court, Appellate Division while Brooklyn hosts the Second Department. There are also several extrajudicial administrative courts, which are executive agencies and not part of the state Unified Court System.\n",
      "Uniquely among major American cities, New York is divided between, and is host to the main branches of, two different U.S. district courts: the District Court for the Southern District of New York, whose main courthouse is on Foley Square near City Hall in Manhattan and whose jurisdiction includes Manhattan and the Bronx; and the District Court for the Eastern District of New York, whose main courthouse is in Brooklyn and whose jurisdiction includes Brooklyn, Queens, and Staten Island. The U.S. Court of Appeals for the Second Circuit and U.S. Court of International Trade are also based in New York, also on Foley Square in Manhattan.\n",
      "\n",
      "\n",
      "=== Politics ===\n",
      "The present mayor is Eric Adams. He was elected in 2021 with 67% of the vote, and assumed office on January 1, 2022.\n",
      "The Democratic Party holds the majority of public offices. As of April 2016, 69% of registered voters in the city are Democrats and 10% are Republicans. New York City has not been carried by a Republican  presidential election since President Calvin Coolidge won the five boroughs in 1924. A Republican candidate for statewide office has not won all five boroughs of the city since it was incorporated in 1898. In 2012, Democrat Barack Obama became the first presidential candidate of any party to receive more than 80% of the overall vote in New York City, sweeping all\n",
      "\n",
      "> Evaluation Result:  YES\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    f\"> Response: {response_vector}\\n\\n\"\n",
    "    f\"> Source: {response_vector.source_nodes[0].source_text}\\n\\n\"\n",
    "    f\"> Evaluation Result: {eval_result}\\n\\n\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b1ccc17-d16f-4440-bdf2-15c83bc81159",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama_index_ravi",
   "language": "python",
   "name": "llama_index_ravi"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
